{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "freelance-nerve",
   "metadata": {},
   "source": [
    "# AttackRecipe\n",
    "`AttackRecipe` aims to find a perturbation of an input text satisfies the attack's goal to fool the given `FlintModel`. In contrast to `Transformation`, `AttackRecipe` requires the prediction scores of the target model.  textflint provides an interface to integrate the easy-to-use adversarial attack recipes implemented based on `textattack`. Users can refer to [textattack](https://github.com/QData/TextAttack) for more information about the supported `AttackRecipe`. This section provides a brief introduction to how to use `AttackRecipe` in textflint.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "sudden-vermont",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('/home/yjc/codes/textrobustness')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acting-uncle",
   "metadata": {},
   "source": [
    "## Using an `AttackRecipe` based on `textattack`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "civil-realtor",
   "metadata": {},
   "source": [
    "1. Define a list of `AttackRecipe` in a python file without defining the specific victim model. For example, we create a `attack_ins.py` file with the following commands:\n",
    "\n",
    "```python\n",
    "from textattack.goal_functions import UntargetedClassification\n",
    "from textattack.search_methods import GreedySearch\n",
    "from textattack.constraints.pre_transformation import RepeatModification, StopwordModification\n",
    "from textattack.transformations import WordSwapWordNet\n",
    "from textflint.generation_layer.attack import Attack # Note that here we use the Attack from textflint\n",
    "\n",
    "# Define the goal function class\n",
    "goal_function = UntargetedClassification\n",
    "# We'll constrain modification of already modified indices and stopwords\n",
    "constraints = [RepeatModification(),\n",
    "               StopwordModification()]\n",
    "# We're going to use WordSwapWordNet as the attack transformation.\n",
    "transformation = WordSwapWordNet()\n",
    "# We'll use the Greedy search method\n",
    "search_method = GreedySearch()\n",
    "# Now, let's make the attack from the 4 components:\n",
    "attack = Attack(goal_function, constraints, transformation, search_method)\n",
    "\n",
    "# ... \n",
    "# many attacks form an attack list\n",
    "attacks = [attack]\n",
    "```\n",
    "\n",
    "2. Define the path of above file in the config json file. For example, the config file `SA.json` might look as follows:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"task\": \"SA\",\n",
    "  \"max_trans\": 1,\n",
    "  \"semantic_validate\": false,\n",
    "  \"semantic_score\": 0.7,\n",
    "  \"fields\": \"x\",\n",
    "  \"keep_origin\": false,\n",
    "  \"return_unk\": true,\n",
    "  \"transformation_config\": {},\n",
    "  \"transformation_methods\": [],\n",
    "  \"subpopulation_methods\": [],\n",
    "  \"attack_methods\": \"attack_ins.py\" //path to attack_ins.py\n",
    "}\n",
    "```\n",
    "\n",
    "3. Load the SA test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "sufficient-kenya",
   "metadata": {},
   "outputs": [],
   "source": [
    "from textflint.input_layer.model.test_model.model_helper import data_loader_csv\n",
    "sa_data_set = data_loader_csv('/home/yjc/codes/textrobustness/textflint/input_layer/model/test_model/train.csv')\n",
    "test_data_set = sa_data_set[int(len(sa_data_set) * 0.7):]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "speaking-mattress",
   "metadata": {},
   "source": [
    "4. Create your own modelwrapper that implementing the function `evaluate` and `encode`. More details can be found in the `modelwrapper` tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "spanish-diving",
   "metadata": {},
   "outputs": [],
   "source": [
    "from textflint.input_layer.model.wrappers.textcnn_torch_wrapper import TextCNNTorchWrapper\n",
    "textcnn_wrapper = TextCNNTorchWrapper()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "breeding-monaco",
   "metadata": {},
   "source": [
    "5. Feeding the dataset `test_data_set`, output path `out_dir_path`, config file `config` and model `textcnn_wrapper` to the SA engine, and run it! textflint will automatically scan the `attack_ins.py` file and load the `attacks` inside. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "aware-reading",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34;1mtextflint\u001b[0m: ******Start load!******\n",
      "100%|██████████| 1544/1544 [00:00<00:00, 2156.11it/s]\n",
      "\u001b[34;1mtextflint\u001b[0m: 1544 in total, 1544 were loaded successful.\n",
      "\u001b[34;1mtextflint\u001b[0m: ******Finish load!******\n",
      "\u001b[34;1mtextflint\u001b[0m: ******Start DoubleDenial!******\n",
      "100%|██████████| 1544/1544 [00:00<00:00, 6363.64it/s]\n",
      "\u001b[34;1mtextflint\u001b[0m: DoubleDenial, original 1544 samples, transform 281 samples!\n",
      "\u001b[34;1mtextflint\u001b[0m: Save samples to /home/yjc/codes/textrobustness/test_result/ori_DoubleDenial_281.json!\n",
      "\u001b[34;1mtextflint\u001b[0m: Save samples to /home/yjc/codes/textrobustness/test_result/trans_DoubleDenial_281.json!\n",
      "\u001b[34;1mtextflint\u001b[0m: ******Finish DoubleDenial!******\n"
     ]
    }
   ],
   "source": [
    "from textflint.engine import Engine\n",
    "from textflint.input_layer.config.config import Config\n",
    "\n",
    "config = Config.from_json_file('/home/yjc/codes/textrobustness/textflint/common/config_files/SA/SA.json')\n",
    "out_dir_path = '/home/yjc/codes/textrobustness/test_result/'\n",
    "\n",
    "engine = Engine('SA')\n",
    "engine.run(test_data_set, out_dir_path, config, textcnn_wrapper)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liquid-baptist",
   "metadata": {},
   "source": [
    "The adverisial samples based on the `AttackRecipe` will be also automatically saved to the directory `out_dir_path`, and we can take a quick look at the contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cubic-andrew",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original:  {\"x\": \" @user \\\"#love is not  , but brings only sadness, because it can not be hold.\\\" erich maria remarque  mountain valley \", \"y\": \"negative\", \"sample_id\": 10}\n",
      "\n",
      "original:  {\"x\": \"#bustyescos  haha really great! @user @user @user \", \"y\": \"positive\", \"sample_id\": 19}\n",
      "\n",
      "transformed:  {\"x\": \"@user \\\" # don't hate is not, but brings only sadness, because it cannot be hold. \\\" erich maria remarque mountain valley\", \"y\": \"negative\", \"sample_id\": 10}\n",
      "\n",
      "transformed:  {\"x\": \"# bustyescos haha really not bad! @user @user @user\", \"y\": \"positive\", \"sample_id\": 19}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('/home/yjc/codes/textrobustness/test_result/ori_DoubleDenial_267.json', 'r') as f:\n",
    "    for ex in f.readlines()[:2]:\n",
    "        print(\"original: \", ex)\n",
    "        \n",
    "with open('/home/yjc/codes/textrobustness/test_result/trans_DoubleDenial_267.json', 'r') as f:\n",
    "    for ex in f.readlines()[:2]:\n",
    "        print(\"transformed: \", ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "blessed-amateur",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In this tutorial, we briefly describe how to use `textattack`'s `AttackRecipe` to generate adverisial samples. We also support loading multiple attacks at once and executing them all by simply runing the `engine`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "operating-device",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
